Recent progress in deep generative models has been fueled by two paradigms -- autoregressive and adversarial models. We propose a combination of both approaches with the goal of learning generative models of text. Our method first produces a high-level sentence outline and then generates words sequentially, conditioning on both the outline and the previous outputs. We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders. This provides strong, informative conditioning for the autoregressive stage. Our quantitative evaluations suggests that conditioning information from generated outlines is able to guide the autoregressive model to produce realistic samples, comparable to maximum-likelihood trained language models, even at high temperatures with multinomial sampling. Qualitative results also demonstrate that this generative procedure yields natural-looking sentences and interpolations.

Update after author response: It's good to see the efforts of conducting human evaluation, and I encourage the authors to include more details, e.g., how the annotators are selected, what questions are asked, into the next revision. This manuscript proposes a generative adversarial approach for text generation. Specifically, sentences are encoded into vectors in a "latent space" by a RNN encoder; a decoder conditions on the vector and generates text auto-regressively. The latent representations are adversarially regularized towards a fixed prior. The method is evaluated on both unconditional and conditional generation settings, and the paper argues for better performance than baselines.

Recent progress in deep generative models has been fueled by two paradigms -- autoregressive and adversarial models. We propose a combination of both approaches with the goal of learning generative models of text. Our method first produces a high-level sentence outline and then generates words sequentially, conditioning on both the outline and the previous outputs. We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders. This provides strong, informative conditioning for the autoregressive stage.